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Contingent Instructors and Student Outcomes: 

An Artifact or a Fact? 

ABSTRACT 

This study addresses methodological problems surrounding existing research on exposure 
to contingent instructors and student outcomes. By applying non-aggregated and aggregated 
measures of exposure to contingent instructors to the same data, this analysis demonstrates that 
effects of commonly used measures of exposure to contingent instructors have little to do with 
actual contingent instructor effects on student outcomes. Two multi-level approaches — cross- 
classified and multiple membership models — are applied in the single-institution analysis of 
faculty status effect on student outcomes — grades and one-year retention. The analysis showed 
no variability in student retention and a significant variability in grades by faculty characteristics. 
Compared to their tenured and tenure-track peers, contingent instructors are likely to assign 
higher grades, which may lead to lowered levels of academic challenge and student motivation to 
do their best work. 



KEY WORDS: contingent instructors; retention; college grade performance; multilevel models; 
cross-classified models; multiple membership models. 
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INTRODUCTION 

In the view of some prominent commentators, American research universities “have too 
often failed . . . their undergraduate populations” (Boyer Commission 1998, p.5). This concern 
arises in part from the growing reliance of research universities on so-called contingent 
instructors — part-time, non-tenure-track faculty, and graduate teaching assistants — and its 
presumed impact on the quality of instruction. Wellman, Desrochers, and Colleen (2008) report 
that, since 1998, the percentage of spending for the direct cost of instruction — faculty salaries 
and benefits — has declined. This decline reflects a continuing growth in the share of part-time 
and non-tenure track faculty in U.S. postsecondary institutions. 

The most important reason for hiring contingent instructors is budgetary constraints. 
Colleges “can hire up to two dozen part-time faculty for roughly the same amount it costs to hire 
a full-time faculty member" (Stephens & Wright, 1999). Another important advantage is 
providing institutional flexibility. When enrollment drops or changes are introduced in the 
general education curriculum, “the number of part-time faculty is easily adjusted by not 
renewing contracts” (Banachowski, 1997). Departments with a professional orientation also seek 
to hire contingent instructors who possess practical experience and expertise in the area 
(Banachowski, 1997, Haeger, 1998). Finally, sometimes part-time faculty themselves are 
“grateful for being able to teach part-time because of the prestige and fulfillment it adds to their 
life” (Reed, 1985, cited in Banachowski, 1997). 

Contingent instructors typically lack job stability and may also lack adequate support 
services, office space, benefits, professional development opportunities, and equal pay for equal 
work. Their lower earnings and lack of benefits are “likely to interfere with their work” 
(Benjamin, 2002), and their employment conditions can lead to dissatisfaction (Gappa, Leslie, 



3 

AIR 2010 Forum - Chicago, IL 




Running Head: CONTINGENT INSTRUCTORS AND STUDENT OUTCOMES 



1997). Another important concern surrounding reliance on contingent faculty is related to the 
academic quality of instruction and the overall academic experience of students. “It’s 
conceivable that face-to-face exchanges between students and faculty outside the classroom will 
decline because part-time faculty spend less time on campus and often do not have a designated 
space to meet with students after class” (Kuh, Laird, Umbach, 2004). Thus, employing 
contingent instructors provides financial benefits and flexibility for institutions but may limit 
student-faculty interaction and further affect student outcomes. 

The debate over the relative effectiveness of contingent instructors has led to a growing 
number of studies that explore the effect of exposure to contingent instructors on student 
outcomes. Kehrberg and Turpin (2002) studied the relationship between exposure to part-time 
faculty and first-year retention. The overall negative association between such exposure and 
retention disappeared after controlling for student academic preparation. Ronco and Cahill 
(2004) studied the effect of exposure to contingent instructors and graduate teaching assistants 
on retention, academic achievement and student ratings of instruction. They reported that there is 
“little evidence that instructor type has a widespread impact on student outcomes” (p. 17). 

Counter to these hopeful results, Schibik and Harrington (2004) indicate that “holding academic 
preparation constant, exposure to part-time faculty at levels above 50% during their first 
semester on campus has a direct and significant negative impact on student retention into the 
second semester” (p.5). Jaeger and Hinz (2008) found that as exposure to instruction by part-time 
faculty increases, the odds of being retained decrease. Eagan and Jaeger (2009) studied the 
effects of exposure to part-time faculty in a community college system and later transfer to a 
four- year institution. They concluded that students are less likely to transfer within five years of 
enrolling in the system of community colleges as the proportion of credits a student took with 
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part-time faculty increases. Based on the association between graduation rates and the use of 
non-tenure-track faculty (whether part-time or full-time) and, Ehrenberg and Zhang (2004) 
concluded that increase in the shares of either part-time or full-time non-tenure-track faculty 
leads to the reduction of graduation rates. 

The problem with these studies of the effectiveness of contingent instructors is that they 
essentially study the correlation between student outcomes and unspecified student 
characteristics that happen to be correlated with the number and percentage of courses students 
take that are taught by contingent instructors. Findings reported by these studies are likely to be a 
mere statistical artifact. As demonstrated by Bettinger and Long (2005), students who take 
courses from adjuncts differ systematically from students who take courses from tenure-track 
faculty. Student-level aggregation of faculty characteristics may reflect nothing more than 
systematic differences between students who take courses from contingent instructors and 
students who take courses from tenure-track faculty. Similarly, institutions with higher shares of 
contingent instructors are likely to be systematically different from institutions with lower shares 
of such instructors. Hoffmann and Oreopoulos (2006) suggest that a negative relationship 
between student persistence and adjunct usage in institution-level studies could be driven by the 
tendency for schools with higher proportion of adjuncts to have students on the margin of 
dropping out. 

The studies of exposure to contingent instructors continue to emerge, and the 
methodological problems associated with these studies persist. The study presented here applies 
non-aggregated and aggregated measures of exposure to contingent instructors to the same data 
and demonstrates that, when aggregated to the student level, a measure of exposure to contingent 
instructors, or percentage of courses taken from contingent instructors, has little to do with the 
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instructor’s effect on student outcomes. Two multi-level approaches — cross-classified and 
multiple membership models — are discussed for the analysis of how faculty status affects class 
grades, one-year cumulative GPA, and one-year retention. Advantages and limitations of these 
multi-level approaches are pointed out. 

CONCEPTUAL ISSUES 

“Perhaps the most commonly applied theory used today to understand contingent workers 
is social exchange theory” (Connelly and Gallagher, 2004, p.977). In contrast to economic 
exchange, social exchange (Blau, 1964) involves unspecified obligations and requires trust. By 
treating employees fairly, an organization initiates social exchange and encourages employees to 
reciprocate to the organization. However, contingent work relationships are based on an 
“asymmetrical” power balance that favors employers (Beard and Edwards, 1995; see also 
Gallagher, 2002). Contingent workers hold insecure positions with little control and 
predictability and are, therefore, less willing to reciprocate by effort. Umbach’s (2007) findings 
were consistent with this reasoning. Based on data from the Faculty Survey of Student 
Engagement (FSSE), Umbach concludes that contingent faculty interact with students less 
frequently, use active and collaborative techniques less often, spend less time preparing for class, 
and have lower academic expectations for student performance than do their tenured and tenure- 
track peers. These characteristics of contingent faculty conceivably have different and complex 
effects on different student outcomes of interest. 

First, if contingent instructors have lower academic expectations, they are also likely to 
give higher grades. Further, professors frequently believe that grades influence student 
evaluations. Adjunct and part-time faculty are “especially vulnerable to student evaluation 
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pressures” (Kamber 2008b, p.57) and might be expected to give higher grades. If the share of 
classes from contingent instructors measured the effect of contingent instructors, higher shares of 
classes from contingent instructors would lead to higher grade point averages. 

Yet grades are not the only student outcome variable of interest. The quality of faculty- 
student interaction is a central factor in student attrition (Tinto, 1993). If contingent faculty 
interact with students less frequently (Umbach, 2007), one would expect that students who take 
fewer classes from contingent faculty and more classes from tenure-track faculty would be more 
likely to persist, even though exposure to contingent faculty tends to raise students’ grades. 

Two control variables— discipline area and class size— are included at the faculty level in 
the models of class grades. Johnson (2010) indicates that class size has a negative effect on 
students' grades; and models with logarithmic representation of a class size show a superior fit 
thus indicating that the effect of class size diminishes as class size increases. A logarithmic 
representation of class size is incorporated in the models of class grades presented here; and this 
variable is hypothesized to have a negative effect on grade performance. 

Prior studies (Benjamin, 2002; Bettinger and Long, 2005) show that the share of 
contingent instructors varies by discipline. Further, grades vary across disciplines. Therefore, 
discipline areas are included in the models of class grades. The discipline areas are categorized 
based on the Biglan (1973a, 1973b) classification scheme that involves three dimensions: hard 
versus soft, applied versus pure, and life versus non-life. The hard- soft dimension distinguishes 
fields that have attained a paradigmatic status (Kuhn 1962) from fields that have not attained a 
paradigmatic status, or fields that have achieved a high degree of consensus about their 
knowledge and methods and fields that have not achieved such consensus. The applied-pure 
dimension distinguishes between fields that concentrate on creating knowledge (pure) or 
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applying knowledge from other fields (applied). The life-non-life dimension distinguishes 
disciplines by their objects of study: “lifesystems” (life) or non-living objects. (The fields 
associated with each Biglan area can be found in variable descriptions of Table 1.) It is expected 
that grades significantly vary by the faculty area of expertise with fewer high grades in hard 
disciplines. 

At the student level, prior research on the effect of first-year student exposure to 
contingent instructors on student outcomes have typically controlled for gender, ethnicity, age, 
high school GPA, state residency, test scores, and course load (see, for example, Bettinger, 2005; 
Ronco, 2004; Schibik and Harrington, 2004). Because the new freshmen at the study institution 
are traditional students coming directly from high school, age is not included in the models here. 
On the other hand, membership in a fraternity or sorority is included in the models here, since 
about a third of new freshmen at the study institution are members of such social organizations. 
Fraternities and sororities “provide individuals with opportunities to establish repetitive contact 
with other members of the institution in circumstances which lead to the possibility of 
integration” (Tinto, 1988, p.446). Hence, Greek membership is expected to have a significant 
positive effect on retention. At the same time, the author remains agnostic about its effect on 
grade performance. 

METHODOLOGICAL PROBLEMS 

No matter what their results, previous studies of exposure to contingent instructors (e.g., 
Kehrberg and Turpin, 2002; Schibik and Harrington, 2004; Ronco, 2004) have measured 
exposure as the share of courses from contingent instructors divided by the overall number of 
courses a student took. The percentages of courses taught by contingent instructors were 
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frequently grouped in ranges such as 0% to 24%, 25% to 49%, 50% to 74%, and 75% to 100%. 
Such an approach is likely to lead to a statistical artifact (Johnson, 2006). For example, a student 
taking only one class can have either 0-24% exposure or 75%-100% exposure, but not 25%-49% 
or 50%-74% exposure. A student taking two classes cannot have 25%-49% exposure. Table 2 
provides an illustration of the association between the measure of exposure and the number of 
classes a student takes. 

Even if the share of courses taught by contingent faculty is not grouped into faulty 
ranges, it is reasonable to expect a significant correlation between students’ course load and their 
exposure to contingent instructors. To avoid the correlation between the exposure to contingent 
instructors and the number of classes a student takes, one can substitute these two predictors with 
the number of classes from tenure-track faculty and the number of classes from contingent 
instructors. If the hypothesis of equality of coefficients for classes from tenure-track faculty and 
classes from contingent faculty is rejected, one would conclude that there is a significant 
difference in student outcomes depending upon faculty status. However, whether one uses the 
share of courses from contingent instructors or the test of the null hypothesis of the equality of 
regression coefficients, the result will be prone to errors, because both approaches are based on 
the aggregation of student-faculty level to the student level. 

The problems that arise with aggregation are frequently described in literature devoted to 
multi-level analysis. Snijders and Bosker (1999) point out potential errors resulting from 
aggregation: the “shift of meaning” the “ecological fallacy ,” and neglect of the original data 
structure. For example, exposure to contingent faculty is a student-level variable that does not 
necessarily reflect the faculty-student interaction (the shift of meaning). A significant positive 
correlation between exposure to contingent faculty and grade point average does not imply that a 
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student’s grade is likely to be lower in a class taught by tenure-track faculty or vice versa (the 
ecological fallacy). The ecological fallacy is also related to the issue of confounding (see, for 
example, Freedman, 1999): students who take more courses taught by contingent instructors may 
be different from students who take more courses taught by tenure-track faculty in many ways 
besides exposure to contingent instructors. 

As illustrated by Aitkin and Longford (1986), aggregation and neglect of the original data 
structure can be “dangerous at best and disastrous at worst” (p.42). Fig.l illustrates how 
disregard for the original data structure can be misleading in studies of exposure to contingent 
instructors. Based on the left-hand graph of the student-faculty data, student 1 received grades of 
“C” and “B” from tenure-track faculty members and an “A” from a contingent instructor. 

Student 2 received a grade of “C” from a tenure-track faculty member and a grade of “B” from a 
contingent instructor. Student 3 received a grade of “D” from a tenure-track faculty member and 
grades of “C” and “D” from contingent instructors. Overall, the graph on the left-hand side of 
Fig.l clearly indicates that students are more likely to receive higher grades in courses taught by 
contingent instructors. But aggregation of these grades to the student level would result in a 
negative association between exposure to contingent instructors and grade point averages. 

Student 1 has the lowest exposure to contingent instructors (33%) and the highest grade point 
average (3.00). Student 3 has the highest exposure to contingent instructors (67%) and the lowest 
grade point average (1.33). Student 2 is in between with the 50% exposure to contingent 
instructors and the grade point average of 2.50. 

One cannot model student-faculty data as independent observations, since it would lead 
to the “miraculous multiplication of units” (Snijders and Bosker, 1999, p.15). In the example 
presented in Fig. 1, treating student-faculty observations as independent would mean treating 
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three students as eight. This would lead to a serious risk of committing type 1 errors (asserting 
that there is an association, whereas there is no such association in the population). Ideally, one 
would take into account the multi-level data structure of student exposure to contingent faculty. 
The problem is that, while the data structure in studies of student exposure to contingent 
instructors is multi-level, it is not strictly hierarchical. Two models suggested in the multi-level 
literature for non-hierarchical data structures are cross-classified and multiple membership. 

Cross-Classified Model: Class Grade as Dependent Variable 

In a cross-classified model of student exposure to contingent instructors, a grade Y;j is 
nested within the cross-classification of a student i and faculty member j (see Fig. 2). On the one 
hand, students attending classes taught by the same faculty member share some common 
experiences and cannot be treated as independent observations. On the other hand, grades are 
affected by student characteristics; and several observations for one student cannot be treated as 
independent. A cross-classified model (Raudenbush, Bryk, Cheong, and Congdon, 2004; 
Raudenbush and Bryk, 2002) properly treats dependencies among multiple observations related 
to a single faculty member and multiple observations related to a single student and provides 
estimates of the variation introduced by different levels of the hierarchical structure, i.e. the 
extent to which grades of a typical student vary by faculty members and the extent to which 
grades given by a typical faculty member vary by students. The estimation of the cross-classified 
models in this study is carried out using HLM 6, one of the dominant and user-friendly multi- 
level software packages. 

A drawback in using this software to analyze grade data is that the scale of measurement 
of class grades is ordinal, but at the time of writing HLM 6 did not estimate cross-classified 
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models for ordinal dependent variables. Consequently, several models with binomial outcomes 
were specified instead. The three constructed models estimate separately log-odds of 
probabilities of getting a grade of “A,” a grade of “B” or higher, and a grade of “C” or higher. 

The unconditional model (model without predictors) estimating the log-odds of the probability of 
getting an “A”, a “B” or higher, or a “C” or higher is as follows: 



In 



Y„= 1 



v J 



fioij 



(Level 1) 



Where the grade Y,/ is within the cross-classification of student i and instructor j. 

A);, = /oQ +U 0 i +V 0 i 



(Level 2) 



Where y 00 is the intercept, u 0j is a residual (also called random effect) for students and v () j is a 
residual for faculty members. It is assumed that u 0i ~ n((), S 2 ) and v 0 . ~ n( O,^ 2 ). A variance 

component for students S 2 estimates the extent to which the probability of getting a certain grade 
from a typical faculty member varies by student. A variance component for faculty members 
d 2 estimates the extent to which the probability of getting a certain grade by a typical student 
varies by a faculty member. 



Multiple Membership Linear Regression Model: Grade Point Average as Dependent 
Variable 

While there can be multiple class grades per student, there is only one cumulative grade 
point average per student at a given time. This means that the lowest level for grade point 
averages is the student level and that a cross-classified model cannot be applied. Each student 
takes courses from several faculty members, and, therefore, the strictly hierarchical structure of 
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students nested within faculty cannot be applied either. The model that accommodates the data 
structure where lower units belong to several classification units is a multiple membership model 
(Hill and Goldstein, 1998; Rasbash and Browne, 2001; Browne, Goldstein, and Rasbash, 2001). 
Fig. 3 illustrates the difference between data structures for cross-classified and multiple 
membership models. A single arrow represents a single membership classification or a grade 
belonging to one student and one faculty member. A pair of arrows represents multiple 
membership or one student exposure to multiple instructors. 

The outcome variable is the second-semester cumulative grade point average, which is 
defined on the usual 0.00 to 4.00 scale and treated as a continuous variable. The multiple 
membership model of the grade point average can be represented as follows: 

V, = A), (Level 1) 

A) i = A) + z w u li 0j + e ot ( Level 2) 

Where J3 0 is the intercept, u 0j is a residual for instructors and e Qi is a residual for students 

with u 0j ~ a(o,<A) and e 0i ~ a((), cf 2 ). A variance component for faculty members 

Si estimates the extent to which grade point averages vary by faculty members. A variance 

component for students e> 2 estimates the extent to which grade point averages vary by students. 

Weight Wy is attributed to a student-faculty combination and can be either equal for all faculty 

members a student has been exposed to or proportionate to the number of credit hours a student 
took from a given faculty members. The sum of weights should equal to one for each student. 
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Multiple Membership Logit Model: Retention as Dependent Variable 
Because the lowest level for retention is the student level and a student is exposed to 
several faculty members, a multiple membership model also applies here. The dependent 
variable equals one if a student returned for the second year and zero if a student did not return. 
The multiple membership model with a binomial outcome can be represented as follows: 




= fi* 



(Level 1) 



A) 1 =A>+Z v V < o; (Level 2) 

Where /?„ is the intercept and u 0j is a residual for instructors with u j ~ n(o , rC ) . A 

variance component for instructors §1 estimates the extent to which the probability of return by a 
typical student varies by faculty members. 

Currently, only the software package MLwiN has special features for estimating multiple 
membership models with a detailed manual for implementing such models (Browne, 2009). 
MLwiN is an excellent choice for estimating a multiple membership variance decomposition 
model or an unconditional model. However, it does not provide a possibility of adding predictors 
at the faculty level and cannot resolve the issue of aggregation of exposure to contingent faculty 
to the student level. Thus, while being resolved in a cross-classified model, the problem 
associated with aggregation of independent variables to the student level persists in multiple 
membership models. 
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DATA AND DESCRIPTIVE STATISTICS 

The institution studied is a Research University (high research activity) with about 4,000 
first-time freshmen enrolled each fall. Data for 2008 new freshmen are included in the analysis. 
The course information was taken from the fall and spring semesters of the first year of studies. 
The variable descriptions as well as descriptive statistics are presented in Table 1. 

The data for the study include 3,911 observations at the student level, 671 observations at 
the faculty level, and 31,199 student-faculty combinations. If a student took more than one class 
from the same faculty member, the grades from these classes are averaged. Each faculty member 
in the data set can teach more than one class to first-year freshmen. At the same time, one can 
expect that several classes taught to freshmen by the same faculty member are similar in class 
size. Taking into account that class level is not introduced here, an average class size for all 
classes a faculty member taught to new freshmen is included as a proxy of a class size. 

Based on descriptive statistics in Table 1, first-year students take less than one third of 
their courses (2.87 out of 9.37) and credit hours (8.88 out of 27.49) from tenured or tenure-track 
faculty. At the same time, the share of tenured or tenure-track faculty in the data is about 44%. 
The distribution by discipline areas shows that most of the courses taken by first- year students 
come from hard-pure-non-life (22%) and soft-pure-non-life disciplines (38%), which is expected 
because of the prevalence of Science, Mathematics, and Humanities in the university’s Core 
Curriculum. Grade distribution presented in Table 1 shows a substantial percentage of high 
grades: a grade of an “A” constitutes 33% of all grades given to new freshmen during the first 
year of their studies, a grade of “B” constitutes about 34%; and a grade of “C” constitutes about 
18%. The rest of grades included grades of “D” and “F” as well as incompletes and withdrawals. 
A small number of pass-fail grades were excluded. 
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CROSS-CLASSIFIED MODELS OF CLASS GRADES 
Unconditional Models 

The unconditional models presented in Table 3 estimate the extent to which grades vary 
across faculty members and across students. Based on Model 1, the expected probability of 
getting an “A” for a typical student taught by a typical instructor is 0.35 or an “antilogit” of the 

Too -°- 61 

p p 

intercept, y 00 : = — _ ()61 . For a student one standard deviation below the average, the 

1 + e"" 1+ e 

probability of getting an “A” is 0.09 or an “antilogit” of the intercept minus the square root of a 

Too ~^u -0.6 1— 72.83 

p p 

variance component for students, y nn -5 : — = — and for a student one 

r ' 00 " 1 , Yoo-Su . -0.61-7783 

i + e i+e 

standard deviation above the average, the probability of getting an “A” in a typical class is 0.75 
or an “antilogit” of the intercept plus the square root of a variance component for students, 

—0.6 1 ! 7 2.83 

p p 

y m + § u : - +<? = — . Based on ’’antilogits” for faculty, for a faculty member one 

, 1 + e 00+ “ \ + e " M ' 2Si 

standard deviation below the average, the probability of getting an “A” for a typical student is 
0.13; and for a faculty member one standard deviation above the average, the probability of 
getting an “A” for a typical student is 0.66. 

The expected probability of getting a “B” or higher is 0.80 or an “antilogit” of the 
intercept in Model 2 of Table 3. The probability of getting a “B” or higher is 0.43 for a student 
one standard deviation below the average and taught by a typical faculty. The probability of 
getting a “B” or higher is 0.96 for a student one standard deviation above the average and taught 
by a typical faculty. The probability of getting a “B” or higher for a typical student is 0.55 in a 
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class taught by a faculty member one standard deviation below the average and 0.93 in a class 
taught by a faculty member one standard deviation above the average. 

The expected probability of getting a “C” or higher is 0.93 (see Model 3 in Table 3). The 
probability of getting a “C” or higher is 0.74 for a student one standard deviation below the 
average and 0.98 for a student one standard deviation above the average. For a typical student the 
probability of getting a “C” or higher is 0.82 in a class taught by a faculty member one standard 
deviation below the average and 0.97 in a class taught by a faculty member one standard 
deviation above the average. 

The average expected probabilities of getting an “A”, a “B” or higher, and a “C” or 
higher with a one standard deviation range for students and faculty are presented in Fig. 4. 
Differences in grades assigned by faculty members are substantial compared to differences in 
grades received by students. Based on the models in Table 3, differences between faculty 



members account for about 37% 



1.64 



1.64 + 2.83 



of the variance in the model of the log-odds of 



getting an “A”, 34% 



1.48 



1.48 + 2.84 



of the variance in the model of the log-odds of getting a “B’ 



or higher, and 31% 



1.11 



1.11 + 2.60 



of the variance in the model of the log-odds of getting a “C r 



or higher. One should also note here that, taking into account that the population under study 
consists of first-time freshmen, the extent of variation among classes and faculty members is 
expected to be lower than if all undergraduate students were considered. This is due to the fact 
that freshmen are more likely to take similar courses, such as courses required by the university’s 
Core Curriculum, than are seniors who take courses in their respective majors. 
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Conditional Models 

Consistently across models presented in Table 4, a faculty member’s tenure-track status 
has a significant negative impact on the odds of getting higher grades. Based on the models 
without control for the discipline area and class size, the average expected probability of getting 
an “A”, a “B” or higher, and a “C” or higher are 0.39, 0.85, and 0.95 if a grade is given by a 
contingent instructor and 0.26, 0.73, and 0.91, if a grade is given by a tenure-track faculty 
member. (These average expected probabilities are calculated based on Model 1.1, Model 2.1 
and Model 3.1 in Table 4 and sample means in Table 1.) 

After controlling for discipline area and class size, the effect of faculty status on grades 
becomes less substantial, but remains statistically significant. Based on model 1.2 in Table 4 and 
sample means in Table 1, the average expected probability of getting an “A” is 0.39 if a grade is 
given by a contingent instructor and .29 if a grade is given by a tenure-track instructor. The 
average expected probability of getting a “B” or higher is 0.85 if a grade is given by a contingent 
instructor and 0.77 if a grade is given by a tenure track faculty (Model 2.2 in Table 4 and sample 
means in Table 1). The probabilities of getting a grade of “C” or higher are 0.95 and 0.92 for 
contingent and tenure track instructors (Model 3.2 in Table 4 and sample means in Table 1). 
These results also indicate that the discrepancies in the probabilities depending upon a faculty 
status are more substantial for higher grades. 

Models 1.2, 2.2, and 3.2 in Table 4 also show a significant variation in the probability of 
getting a higher grade depending upon class size. For example, a change in class size from 10 to 
100 leads to a decrease in the expected probability of getting an “A” from 0.45 to 0.26 (see Table 
5). A comparison of probabilities in Table 5 also suggests that the effect of class size varies 
depending upon which grade is examined. Class size has a bigger effect on the probability of 
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getting an “A” than on the probability of getting a “B” or higher. And the effect of class size on 
the probability of getting a “B” or higher is more pronounced than the effect of class size on the 
probability of getting a “C” or higher. For instance, a typical student taught by a typical 



instructor is 1.74 



^ 0.45 ^ 



v 0.26 j 



times more likely to get an “A” in a class of 10 than in a class of 100; 



1.17 



^ 0 . 87 ^ 



V 0.75y 



times more likely to get a “B” or higher in a class of 10 than in a class of 100; and 



only 1.03 



^ 0 . 95 ^ 

V 0.93y 



times more likely to get a “C” or higher in a class of 10 than in a class of 100. 



Overall, while the effect of class size on the probability of getting a “C” or higher is statistically 
significant, it is not substantial. This corresponds to prior research (see, for example, Johnson, 
2010) suggesting that the impact of class size is greater on the likelihood of getting a grade of 
“A” than on a grade of “C” or higher. 

The probabilities presented in Table 5 also show a substantial variation in grades 
depending upon a discipline, with lower probabilities of getting higher grades in hard-pure-life 
and hard-pure-non-life disciplines and higher probabilities of getting higher grades in soft- 



applied-life disciplines. Thus, a typical student is 4.13 



^ 0 . 86 ^ 

V 0.21y 



times more likely to receive a 



grade of “A”, 1.59 



f 0.96 ' 

lobT, 



times more likely to get a grade of “B” or higher, and 1.14 



f 0.98 ^ 
v 086 y 



times more likely to receive a grade of “C” or higher from an instructor in soft-applied-life than 
from an instructor in hard-pure-life discipline. 

At the student level, the most interesting finding is related to the effect of Greek 
membership on grade performance. While being a member of a Greek organization does not 
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have a significant effect on the odds of getting an “A,” it does have a significant effect on the 
odds of getting a “B” or higher and a “C” or higher. The probability of getting a “B” or higher 
for a typical student taught by a typical instructor is 0.82. This probability is 0.81 for a non- 
member and 0.83 for a member of a Greek organization. The probability of getting a “C” or 
higher for a typical student taught by a typical faculty member is 0.94. This probability is 0.94 
for non-members and 0.95 for members of Greek organizations. A positive effect of Greek 
membership could be associated with minimum grade-point standards required for continued 
Greek membership. 

The effects of other student-level characteristics do not hold surprises. Based on Models 
1.2, 2.2, and 3.2 in Table 4, female students are significantly more likely to get higher grades. 
Thus, the expected probability of getting an “A” for a typical student taught by a typical faculty 
is 0.34. This probability is 0.40 for female students and 0.29 for male students. The probability 
of getting a “B” or higher is 0.85 for female students and 0.78 for male students. The probability 
of getting a “C” or higher is 0.95 for females, and 0.93 for males. 

According to the expected probabilities by ACT scores in Table 5, students who come to 

( 0.55^1 

the study university with the ACT score of 30 are, on average, 4.04 ^ ^ times more likely to 



( 0.9H ( 0.97 

receive an “A”, 1.51 times more likely to receive a “B” or higher, and 1.10 

l 0.60 J l 0.88 



times more likely to receive a “C” or higher in their first-year courses than students who come 



with the ACT score of 20. Students with 4.00 high school GPA are 3.43 — — times more 

l 0.14 J 
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likely to receive an “A”, 1.59 



f 0.89 ^ 
, 056 , 



times more likely to receive a “B” or higher, and 1.16 



f 0.97 
v 083 y 



times more likely to receive a “C” or higher than students with 3.00 high school GPA. 



The cross-classified conditional models presented here show that the instructor’s tenure- 
track faculty status has a significant negative effect on a student’s probability of getting higher 
grades; and this negative effect remains significant after control for discipline area and an 
average class size for an instructor. Such faculty-level characteristics as discipline area and an 
average class size also show a significant association with student grade performance. 



MULTIPLE MEMBERSHIP MODELS OF PERSISTENCE AND GRADE POINT AVERAGE 
Unconditional Models 

The unconditional single-level and multiple membership models are presented in Table 6. 
As mentioned before, in multiple membership models we assign weights for each pairing of a 
faculty and a student. These weights sum to 1 for each student. In multiple membership models 
of exposure to contingent faculty, one can explore at least two ways to attribute weights to 
instructors. On the one hand, one can assign a weight that is proportional to the number of credit 
hours associated with a student-faculty combination. On the other hand, one can assign an equal 
weight to each faculty member. Based on the unconditional models presented in Table 6, one 
can, on the one hand, test the null hypothesis that the faculty-level random effect is zero, 

H q : u QJ = 0 , and, on the other hand, define whether equal or proportional weights model better 
fits the data. 
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The null hypothesis of a zero faculty-level random effect can be tested using Deviance, or 
a measure of lack of fit between a model and data 1 . The larger the Deviance, the poorer the fit. 
The difference between the Deviance for a model without a faculty-level random effect and the 
Deviance for a model with faculty-level variance component has a x " distribution with the 
degrees of freedom equal to the difference in the number of parameters estimated. Deviance 
Information Criterion (DIC) adjusts for the number of parameters in the model and can be 
compared directly, without taking into account the degrees of freedom. A smaller DIC suggests a 
better model (Browne 2009, 28-33). 

A comparison of the unconditional models of retention in Table 6 shows that adding 
faculty-level variance component did not lead to significant changes in Deviance and Deviance 
Information Criterion. Proportional weights and equal weights models also show close to zero 
and insignificant faculty-level variance component. Based on ’’antilogits” for a proportional 
weights model, for a faculty member one standard deviation below the average, the probability 
of return for a typical student is 0.87 (or, more precisely, 0.868); and for a faculty member one 
standard deviation above the average, the probability of return for a typical student is 0.87 (or, 
more precisely, 0.872). In other words, no variability in log-odds of retention is found depending 
upon faculty members students are exposed to. At the same time, important variability exists in 
student grade point averages depending upon faculty members teaching the courses they took. 

Based on DIC comparison for models of spring 1 cumulative GPA in Table 6, an equal 
weights model has a superior fit. The following interpretations are based on the equal weights 
model. Based on intra-class correlation, the proportion of the total variance in grade point 
averages which can be attributed to between-faculty differences is substantial - about 0.76: 
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P = 



Si 0 1-72 

te+<y;,)Uo-53+i.72) 



0.76. 



Fig. 5, a so-called caterpillar plot, provides a visual aid in assessing the statistical significance of 
differences in mean grade point averages by faculty members. There are 553 residuals plotted, 
each representing a single faculty member.” (Residuals represent departures from the overall 
mean for each faculty member, i.e. u Qj .) Faculty members are ranked based on the magnitude of 

the residual. Goldstein and Healy (1995) recommend this type of presentation to judge statistical 
significance based on non-overlapping confidence intervals. They indicate that, while “[i]t is a 
common statistical misconception to suppose that two quantities whose 95% confidence intervals 
fail to overlap are significantly different at the 5% level” (p. 175), one can adjust the confidence 
level to be able to use the non-overlap criterion. The width of the intervals to achieve a 5% 
significance level should be ± 1.396 x 8 , and this width is used in Fig. 5. While the proportion of 
the total variance in grade point average explained by between-faculty differences is substantial, 
it is clear from the graph that there is a considerable overlap of intervals, so that only relatively 
widely separated faculty members can be judged as having significantly different residuals. 3 

No significant variation in log-odds of retention was found depending upon which faculty 
members students are exposed to. At the same time, significant variability depending upon 
instructors was found in student grade point averages. A comparison between proportional 
weights and equal weights multiple membership models for GPA as an outcome showed a 
superior fit of the equal weights models. The next step is increasing the complexity of the model 
by adding predictors. 
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Conditional Models 

The unconditional model of student retention shows that there is no significant amount of 
variability in the intercept across instructors. Taking this into account, the faculty-level random 
effect, u Qj , can be removed from the equation. Absence of significant variability across 

instructors also implies that in the study presented here, retention is sufficiently explained by 
student-level characteristics. (Testing the variability in the slope coefficients is beyond the scope 
of this study.) This leads to the conclusion that the effect of faculty status and other faculty 
characteristics on the log-odds of second-year return is not significant. 

The unconditional model of grade point averages showed significant between-faculty 
variability. The next step is to explain this variability by faculty-level variables. Based on the 
results of the cross-classified models presented above, one can already ascertain that course 
grades are related to the faculty status of the instructor — tenure-track or non-tenure-track. The 
analysis presented below will demonstrate that the association between the student-level measure 
of exposure to contingent faculty and grade point average cannot be used to make inferences 
about a student grade being lower in a class taught by tenure-track faculty or vice versa. 

Because the equal weights unconditional model showed a superior fit, the subsequent 
conditional models will be equal weights models with exposure to contingent instructors being 
measured by classes as opposed to credit hours. (While not presented here, the analysis based on 
credit hours was also carried out and showed results substantively similar to the results presented 
here, but a slightly inferior model fit.) 

In view of a prior finding of a negative association between the tenure-track faculty status 
and the probability of getting higher grades, one would expect a positive association between the 
proportion of courses taught by contingent instructors and grade point average. Two approaches 
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are used to measure the effect of contingent instructors on student GPA in Table 7. One of these 
approaches follows prior research on exposure to contingent instructors and uses the index of 
exposure or the number of classes taught by contingent instructors divided by the overall number 
of classes a student took. The other approach is based on estimating the regression coefficients of 
the number of classes taught by tenure-track instructors and the number of classes taught by 
contingent faculty and testing the null hypothesis that these regression coefficients are equal. The 
equality of these regression coefficients is tested using Wald test (Rasbash, Steele, Browne and 
Goldstein, 2009, pp. 125-126) of the null hypothesis H 0 : /?, - =0 where /?, and j8 2 are the 

coefficients for the number of classes taught by tenure-track faculty and number of classes 
(taught by contingent instructors respectively. 

Model 1.1 and Model 1.2 of Table 7 do not agree with findings of a cross-classified 
model. Based on Model 1.1, exposure to contingent instructors has a significant negative effect (- 
0.47) on cumulative GPA. The more classes are taught by contingent instructors, the lower the 
cumulative GPA is. Based on Model 1.2, the coefficient for the number of classes taught by 
tenure-track faculty (0.20) is greater than the coefficient for the number of classes taught by 
contingent instructors (0.14), and the null hypothesis on the equality of regression coefficients 
can be rejected based on the Wald test: / 2 (1) = 18.60 , significant at the 5% alpha level. Based 
on these results alone, one would conclude that the exposure to contingent instructors has a 
negative impact on student grades. Yet a comparison of findings based on the cross-classified 
model and multiple membership model with aggregated measure of exposure to contingent 
instructors reflects the worst-case- scenario provided in a hypothetical example of Fig. 1 and 
clearly demonstrates that aggregation in this case is misleading. 
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The explanation for higher grade point averages for students who are more exposed to 
tenure-track faculty can be related to correlation between the measures of exposure to contingent 
instructors and certain student characteristics. Bettinger and Long (2005) demonstrate that 
students with higher ACT scores are less likely to take classes taught by adjuncts during their 
first semester and provide plausible reasons for this. Thus, some universities allow students with 
high ACT scores to register for classes sooner than other students and these students choose 
classes taught by tenure-track instructors. Sometimes departments assign adjuncts to classes at 
times of the day that are less likely to attract typical high performing students. Tenure-track 
faculty members are more likely to teach Honors classes at the study institution, and the 
significant positive correlation between the number of courses taught by tenure track faculty and 
grades can be expected. A further exploration reveals that there are correlations, significant at the 
1% alpha level, between students’ high school GPA and the number of hours or courses taught 
by tenure-track faculty and by contingent faculty. For example, the correlation between the high 
school GPA and the number of courses taught by tenure-track faculty is positive (0.130) and 
significant at the 5% alpha level, while the correlation between the high school GPA and the 
number of courses taught by contingent faculty is negative (-0.124) and significant at the 5% 
alpha level. The correlation between ACT test scores and number of classes taught by tenure- 
track faculty is significant and positive (0.279), while the correlation between test scores and 
number of classes taught by contingent instructors is significant and negative (-0.284). 

After controlling for high school GPA and ACT scores in Models 2.1 and 2.2 in Table 7, 
the effect of exposure to contingent instructors becomes insignificant at the 5% alpha level 
(Model 2.1), and the difference between the effects of number of classes or hours taught by 
tenure track faculty and number of hours taught by contingent faculty ceases to be significant at 
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the 5% alpha level: j 2 (1) = 0.17 . Thus, the negative effect of exposure to contingent instructors 
in Models 1.1 and 1.2 of Table 7 was a mere reflection of class taking behaviors of high 
performing students. 

While the effects of aggregated measures of exposure to contingent instructors on 
cumulative GPA in multiple membership models in Table 7 do not align with the effects of 
tenure-track status of faculty members on probability of getting higher grades in cross-classified 
models in Table 4, the effects of student-level characteristics, such as number of classes a student 
took, gender, ethnicity, state residency, high school GPA, ACT test, and Greek membership, 
have consistent signs and statistical significances. Based on the Equal Weights cumulative GPA 
Model 2 in Table 8, female students have higher grades; so do students who take more classes. 
The effects of ethnicity and state residency on cumulative GPA are not significant. The effects of 
high school GPA, ACT and Greek membership are positive and significant. 

GPA Model 1 in Table 8 shows that, after control for other student-level characteristics 
(gender, ethnicity, state residency, high school GPA, ACT, and Greek membership), the effect of 
exposure to contingent instructors on cumulative GPA remains insignificant. The regression 
coefficients for the number of classes taught by contingent instructors and the number of classes 
taught by tenure-track faculty are practically identical. And, the null hypothesis on the equality 
of these regression coefficients cannot be rejected based on the Wald test: % 2 ( 1) = 0.14 , not 
significant at the 5% alpha level. Finally, the model that combines these two independent 
variables into one, the overall number of classes a student took, shows a superior fit based on 
Deviance and Deviance Information Criterion. 

A comparison of the logistic regression coefficients for the number of classes taught by 
contingent instructors (0.58) and the number of classes taught by tenure-track faculty (0.57) in 
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single-level model retention model in Table 8 shows that these coefficients are very similar. The 
null hypothesis on the equality of these regression coefficients cannot be rejected based on the 
Wald test: (1) = 0. 10 , not significant at the 5% alpha level. And, the single-level retention 

Model 2 that combines these two independent variables into one— the overall number of classes a 
student took— shows a superior fit based on Deviance and Deviance Information Criterion. 

The results of the retention model 2 in Table 8 indicate that the odds of return increase 
1.79 (e 058 ) times with each additional class a student takes, which, on the one hand, could reflect 
the well documented association between part-time enrollment and retention. On the other hand, 
this association can be related to the fact that students who took more classes should have been 
enrolled both in fall and spring semesters, while those who took fewer classes could have been 
enrolled in fall only and dropped out before the spring semester. State residents have 2.12 ( e 075 ) 
times higher odds of return compared to non-residents. The increase of high school GPA by 1 
leads to 1.95 (e 0 67 ) increase in odds of return; and the increase of ACT score by 5 leads to 1.22 
(e 5x0 '04 ) time increase in odds of return. Members of Greek organizations have 2.64 ( e 0 ' 97 ) times 
higher odds of return compared to non-Greeks. The effects of gender and ethnicity are not 
significant at the 5% alpha level. 

LIMITATIONS 

The study has several limitations. First, the analysis is based on data from a single 
moderately large research institution. Types of contingent instructors as well as the effect of 
contingent instructors on student outcomes may vary across institutions; and the findings 
presented here might not apply to other institutions and institution types. 
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Second, while a non-proportional or partial proportional odds model is a better alternative 
for student grades as outcomes, several models with binomial outcomes — a grade of “A”, a grade 
of “B” or higher, and a grade of “C” or higher — were analyzed instead. At the time of writing, 
non-proportional or partial proportional odds cross -classified models were not estimated in 
HLM. 

A third limitation is that class size in the cross-classified model was aggregated to the 
faculty level. While instructors are likely to teach first-year courses or Core Curriculum courses 
of similar sizes, a more accurate model specification with grades nested within students and 
classes and classes nested within faculty members should be considered in future research. 

IMPLICATIONS 

The analysis presented here illustrates that the index of exposure to contingent instructors 
used in many prior studies of instructor effectiveness is liable to errors resulting from 
aggregation. Such errors are widely discussed in the literature devoted to multi-level analysis and 
include the shift of meaning, the ecological fallacy, and the neglect of the original data structure. 

In the present study, a cross-classified model of student grades — a model with grades 
nested within student and faculty members — shows that students are more likely to have higher 
grades in courses taught by contingent instructors. At the same time, when faculty status (tenure- 
track versus contingent) is aggregated to the student level, the effect of exposure to contingent 
instructors on grade performance becomes negative. After controlling for high school academic 
performance, the effect of exposure to contingent instructors becomes insignificant. Thus, the 
absence of association between the aggregate measure of exposure to contingent instructors does 
not mean that there are no differences in grades given by tenure-track and contingent faculty. 
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Studies that suffer from errors of aggregation continue to emerge and sometimes lead to 
strong messages. For example, Glenn (2008) suggests that one should keep adjuncts away from 
introductory courses. This message was based on the study of Eagan and Jaeger (2008) who 
used the percentage of introductory courses with graduate students and the percentage of 
introductory courses with other part-time faculty as a student-level measure of exposure to 
different instructor types. A comparison of aggregated and non-aggregated results provided here 
warns researchers that the percentage of courses taught by contingent instructors should not be 
used to measure the effect of contingent instructors on student outcomes. 

If a student grade is the dependent variable, it is methodologically appropriate to use a 
cross-classified model and incorporate faculty status — contingent or tenure-track — at the faculty 
level. However, the cross-classified model cannot be used when the dependent variable is at the 
student level and not at the cross-classification of student and faculty, which is the case with 
student retention and grade point average. A hierarchical two-level model would not be 
appropriate either, since each student can be exposed to more than one faculty member. 

Effects on a student-level response variable, such as retention, with each student being 
exposed to multiple faculty members, can be conceptualized within the framework of multiple 
membership models. Multiple membership models are useful when the researcher is interested in 
variance decomposition. At the same time, adding predictors from the faculty level is 
problematic. Thus, while multiple membership models take into account the original data 
structure, these models do not provide means for testing the fixed effects from the level of 
multiple membership, such as faculty status. 

The unconditional model of retention presented here showed no variability in log-odds of 
retention depending upon faculty members students are exposed to. Hence, no evidence suggests 
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that faculty characteristics, including faculty rank or status, affect the probability of student 
return for the second year at the study institution. At the same time, the unconditional model of 
grade point averages suggests a significant variation by faculty members. 

Overall, the models presented here indicate that at the study institution the contingent 
status of instructors does not affect student probability of return for the second year. At the same 
time, grades are affected by instructor type. Controlling for other faculty-level and student-level 
characteristics, contingent instructors give higher grades. Since professors who give easy grades 
are believed to have higher teaching evaluations, it is plausible that contingent instructors are 
more hesitant to give low grades to assure that they will continue to teach after the current 
contract expires. 

Grade inflation can lead to lowered levels of academic challenge and student motivation 
to do their best work. Possible ways to deal with the disparity between grading practices by 
tenure-track faculty and contingent instructors should be discussed at the institutional level. 
Strategies that can be implemented include: adopting caps that limit the proportions of “A” 
grades that can be awarded; weighting student evaluations of instruction on the basis of grade 
distribution (Kamber, 2008a); or providing training and support to help adjunct faculty abide by 
grading standards used by tenure-track faculty (Sonner, 2000). 
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END NOTES 



1 A cross-classified model is estimated in HLM using restricted maximum likelihood method. 
One practical consequence is that the likelihood ratio test or Deviance is not available 
(Raudenbush, Bryk, Cheong, and Congdon, 2004, p.ll). Multiple membership models, on the 
other hand, are estimated in MLwiN using Markov Chain Monte Carlo (MCMC) methods 
(Browne, 2009); and l ik elihood ratio tests are available. 

“ The data structure for multiple membership models has one observation per student. For each 
student in the data set there is an information for up to 12 instructors they were taught by in 
columns faculty 1, faculty 2, ..., faculty 12. The corresponding information about weights is 
included in columns weight 1, weight 2, ... , weight 12 for proportional weights and ewl, ew2, 
..., ewl2 for equal weights. There are 671 instructors in the data set, but only 553 appeared in 
the first column. MLwiN plot is based on the instructors that appeared in the first column. 

1 Based on a similar plot with the width of intervals of 1.96, one can also gauge groups of faculty 
members at the lower and upper end of the plot where the confidence intervals for their residuals 
do not overlap zero, which would mean that these are the faculty members that differ from the 
average at the 5% alpha level. 
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TABLE 1. Variable Description and Descriptive Statistics 



Variable 


Variable description 


Mean (SD) 


Level-1 






Course credit hours 


Number of hours a student took from a faculty member 


3.30 (0.83) 


Grade "A" 


1 for a grade of "A"; 0 otherwise 


0.33 (0.47) 


Grade "B" or higher 


1 for a grade of "B” or higher; 0 otherwise 


0.67 (0.47) 


Grade "C" or higher 


1 for a grade of "C" or higher; 0 otherwise 


0.85 (0.36) 


# of level-1 observations 




31,199 


Level-2: Student characteristics 




Female 


1 if female; 0 otherwise 


0.52 (0.50) 


Non-Caucasian 


1 if ethnicity is Caucasian; 0 otherwise 


0.12(0.32) 


State resident 


1 if state resident; 0 otherwise 


0.59 (0.49) 


High School GPA 


High School GPA 


3.69 (0.43) 


ACT or SAT 
Equivalent 


ACT or SAT Equivalent 


25.87 (3.48) 


Greek 


1 if a member of a fraternity or sorority; 0 otherwise 


0.34 (0.47) 


Number of courses 


Number of courses a student took during the fall and 
spring semester of the first year of studies 


9.39(1.45) 


Number of hours 


Number of hours a student took during the fall and spring 
semester of the first year of studies 


27.49 (3.66) 


Courses taught by 
tenure track faculty 


Number of courses taught by tenure track faculty 


2.87(1.60) 


Courses taught by 
contingent instructors 


Number of courses taught by contingent instructors 


6.52(1.89) 


Credit hours taught by 
tenure track faculty 


Number of credit hours taught by tenure track faculty 


8.88(4.91) 


Credit hours taught by 
contingent instructors 


Number of credit hours taught by contingent instructors 


18.61 (5.23) 


Spring 1 Cumulative 


Cumulative GPA for fall and spring semester of the first 




GPA 


year of studies 


2.80 (0.85) 


Retention 


1 if returned; 0 otherwise 


0.87 (0.34) 


# of student-level observations 


3,911 


Level-2: Faculty characteristics 




Tenure-track 


1 if tenure -track; 0 otherwise 


0.44 (0.50) 


Hard-pure-life 


1 if Biological Sciences; 0 otherwise 


0.03 (0.18) 


Hard-pure-non-life 


1 if Chemistry, Geology, Mathematics, Statistics, 
Physics, or General Sciences; 0 otherwise 


0.22 (0.41) 


Hard-applied-life 


1 if Agriculture, Aquaculture, Forestry, Nutrition, 
Pharmacy, or Veterinary Medicine; 0 otherwise 


0.04 (0.19) 


Hard-applied-non-life 


1 if Engineering or Computer Science; 0 otherwise 


0.07 (0.25) 


Soft-pure-life 


1 if Social Sciences; 0 otherwise 


0.08 (0.28) 


Soft-pure-non-life 


1 if Humanities; 0 otherwise 


0.39 (0.49) 


Soft-applied-life 


1 if Education, Family Studies, Communication 
Disorders, or Nursing; 0 otherwise 


0.08 (0.28) 


Soft-applied-non-life 


1 if Economics, Business, or Architecture; 0 otherwise 


0.09 (0.29) 


Log (class size) 


Natural logarithm of average class size for a faculty 
member 


3.46 (0.72) 


# of faculty-level observations 




671 
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TABLE 2. Student Course Load and Exposure to Contingent Instructors 



Course Load 




Exposure to Contingent Instructors 




0%-24% 


25%-49% 


50%-74% 


75%-100% 


One course 


+ 


— 


— 


+ 


Two courses 


+ 


— 


+ 


+ 


Three courses 


+ 


+ 


+ 


+ 
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TABLE 3. Cross-Classified Unconditional Models 





Model 1: 
Grade “A” 


Model 2: 
Grade “B” or 
higher 


Model 3: 
Grade “C” or 
higher 


Intercept J3 0 (SE(J3 0 )) 


-0.61 (0.06)* 


1.42 (0.06)* 


2.60 (0.06)* 


Student (s* ) 


2.83 


2.84 


2.42 


Faculty (c \: ) 


1.64 


1.48 


1.11 



*Significant at the 5% level 
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TABLE 4. Cross-Classified Conditional Models 





Grade 
Model 1.1 


“A” 

Model 1.2 


Grade “B” 
Model 2.1 


or higher 
Model 2.2 


Grade “C” 
Model 3.1 


or higher 
Model 3.2 


Intercept 


-13.60 (0.37)* 


-9.49 (0.52)* 


-11.80 (0.37)* 


-8.53 (0.52)* 


-9.61 (0.41)* 


-7.65 (0.56)* 


Level- 1 














Course credit hours 




-0.16(0.03)* 




-0.08 (0.03)* 




-0.02 (0.03) 


Level-2: Student characteristics 














Female 


0.50 (0.06)* 


0.51 (0.06)* 


0.46 (0.06)* 


0.46 (0.06)* 


0.45 (0.07)* 


0.45 (0.07)* 


Non-Caucasian 


0.07 (0.09) 


0.07 (0.09) 


-0.06 (0.09) 


-0.06 (0.09) 


-0.13(0.09) 


-0.13 (0.10) 


State resident 


0.07 (0.06) 


0.07 (0.06) 


0.06 (0.06) 


0.06 (0.06) 


0.01 (0.06) 


0.01 (0.06) 


High School GPA 


1.70 (0.07)* 


1.72 (0.07)* 


1.82 (0.07)* 


1.83 (0.07)* 


1.74 (0.08)* 


1.75 (0.08)* 


ACT or SAT Equivalent 


0.20 (0.01)* 


0.20 (0.01)* 


0.19(0.01)* 


0.19(0.01)* 


0.14(0.01)* 


0.14(0.01)* 


Member of a fraternity or 


0.03 (0.06) 


0.03 (0.06) 


0.15(0.06)* 


0.15 (0.06)* 


0.28 (0.07)* 


0.27 (0.07)* 


sorority 














Number of classes took 


0.14(0.02)* 


0.14(0.02)* 


0.18(0.02)* 


0.18 (0.02)* 


0.24 (0.02)* 


0.24 (0.02)* 


Level-2: Faculty characteristics 














Tenure-track 


-0.61 (0.12)* 


-0.44 (0.12)* 


-0.72 (0.12)* 


-0.54 (0.11)* 


-0.65 (0.12)* 


-0.52(0.11)* 


Hard-pure-life 




-3.14(0.38)* 




-2.87 (0.37)* 




-2.23 (0.38)* 


Hard-pure-non-life 




-2.88 (0.26)* 




-2.77 (0.26)* 




-2.33 (0.28)* 


Hard-applied-life 




-1.81 (0.41)* 




-1.06 (0.40)* 




-0.52 (0.43) 


Hard-applied-non-life 




-2.18(0.32)* 




-1.47 (0.32)* 




-1.08 (0.34)* 


Soft-pure-life 




-2.43 (0.30)* 




-1.70 (0.31)* 




-1.10(0.32)* 


Soft-pure-non-life 




-2.69 (0.24)* 




-1.70 (0.25)* 




-1.05 (0.27)* 


Soft-applied-non-life 




-2.76 (0.29)* 




-1.78 (0.30)* 




-1.05 (0.32)* 


ln(CS) 




-0.37 (0.08)* 




-0.37 (0.08)* 




-0.19(0.08)* 


Random parameters 














Student (<5 2 ) 


1.74 


1.77 


1.81 


1.83 


1.81 


1.84 


Faculty (<3 2 ) 


1.81 


1.30 


1.59 


1.08 


1.26 


0.84 



*Significant at the 5% level 
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TABLE 5. Expected Probabilities of Getting an “A”, a “B” or Higher, and a “C” or Higher 



by Class Size, Discipline Area, High School GPA, and ACT Scores 





Grade "A" 

(Model 1.2 in Table 4) 


Grade "B" or higher 
(Model 2.2 in Table 4) 


Grade "C" or higher 
(Model 3.2 in Table 4) 


Expected probabilities by class size 






Class of 10 


0.45 


0.87 


0.95 


Class of 20 


0.38 


0.84 


0.95 


Class of 50 


0.31 


0.79 


0.94 


Class of 100 


0.26 


0.75 


0.93 


Expected probabilities by discipline 






Hard-pure-life 


0.21 


0.61 


0.86 


Hard-pure-non-life 


0.25 


0.63 


0.85 


Hard- applied-life 


0.50 


0.90 


0.97 


Hard-applied-non-life 


0.41 


0.86 


0.95 


Soft-pure-life 


0.35 


0.83 


0.95 


S oft-pure-non-life 


0.29 


0.83 


0.95 


Soft-applied-life 


0.86 


0.96 


0.98 


Soft-applied-non-life 


0.28 


0.82 


0.95 


Expected probabilities by high school GPA 






High school GPA 2.50 


0.06 


0.34 


0.67 


High school GPA 3.00 


0.14 


0.56 


0.83 


High school GPA 3.50 


0.27 


0.76 


0.92 


High school GPA 4.00 


0.47 


0.89 


0.97 


Expected probabilities by ACT or SAT Equivalent 






ACT 20 


0.14 


0.60 


0.88 


ACT 25 


0.30 


0.79 


0.94 


ACT 30 


0.55 


0.91 


0.97 


ACT 35 


0.77 


0.96 


0.98 
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TABLE 6. Multiple Membership Unconditional Models 





Single-level 


Proportional 

weights 


Equal weights 


Outcome: Spring 1 Cumulative GPA 


Intercept J3 Q (SE(/3 Q )) 


2.80 (0.01)* 


2.99 (0.07)* 


2.96 (0.06)* 


Student-level A 2 0 {se{s^ 0 )) 


0.72 (0.02)* 


0.54 (0.01)* 


0.53 (0.01)* 


Faculty-level S^ 0 {se{s^ 0 )) 




1.66 (0.17)* 


1.72 (0.17)* 


Deviance (MCMC) 


9827.29 


8663.78 


8613.45 


Deviance Information Criterion (DIC) 


9829.29 


8972.23 


8945.35 


Outcome: Retention to the second year 


Intercept J3 Q (SE(/3 Q )) 


1.89 (0.05)* 


1.90 (0.05)* 


1.89 (0.05)* 


Faculty-level A; 0 (se(s* 0 )) 




0.02 (0.02) 


0.00 (0.00) 


Deviance (MCMC) 


3040.59 


3037.61 


3040.50 


Deviance Information Criterion (DIC) 


3041.53 


3039.66 


3041.52 
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TABLE 7. Multiple Membership Equal Weights Models of Spring 1 Cumulative GPA with 



and without Control for High School Academic Performance 





Model 1.1 


Model 1.2 


Model 2.1 


Model 2.2 


Outcome: Spring 1 Cumulative GPA 
Intercept J3 Q (SE(J3 Q )) 


1.67 (0.15)* 


1.33 (0.12)* 


-2.40 (0.19)* 


-2.35 (0.16)* 


Number of classes/Number of 
classes taught by tenure-track 


0.16(0.01)* 


0.20 (0.01)* 


0.14(0.01)* 


0.15 (0.01)* 


faculty /?, iSE(/3 , )) 

Share of classes/Number of classes 
taught by contingent instructors 


-0.47 (0.15)* 


0.14(0.01)* 


0.06 (0.12) 


0.14(0.01)* 


P 2 (SE(P 2 )) 

High School GPA p 3 (SE(p 3 )) 
ACT P 4 (SE(P 4 )) 
Student-level 8] a (se(s^ 0 )) 


0.50 (0.01)* 


0.49 (0.01)* 


0.71 (0.03)* 
0.05 (0.00)* 
0.42 (0.01)* 


0.71 (0.03)* 
0.05 (0.00)* 
0.42 (0.01)* 


Faculty-level 8p (SE(sp )) 


1.66 (0.17)* 


1.64 (0.16)* 


0.75 (0.10)* 


0.75 (0.09)* 


Chi-square (df=l), H 0 : P x — P 2 = 0 
Deviance (MCMC) 


8347.58 


18.60* 

8343.02 


7733.16 


0.17 

7732.12 


Deviance Information Criterion 
(DIC) 


8684.17 


8678.70 


7992.75 


7992.28 



*Significant at the 5% alpha level. 

Model 1.1: Number of classes taken by a student and index of exposure to contingent instructors as predictors, no 
control for high school academic performance 

Model 1.2: Number of classes taught by tenure-track instructors and number of classes taught by contingent 
instructors, no control for high school academic performance 

Model 2.1: Number of classes taken by a student and index of exposure to contingent instructors as predictors with 
control for high school academic performance 

Model 2.2: Number of classes taught by tenure-track instructors and number of classes taught by contingent 
instructors with control for high school academic performance 
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TABLE 8. Single-Level Retention Model and Multiple Membership Cumulative GPA 

Model 









Multiple 


Multiple 




Single-Level 


Single-Level 


Membership 


Membership 




Retention 


Retention 


Equal Weights 


Equal Weights 




Model 1 


Model 2 


Cumulative 


Cumulative 








GPA Model 1 


GPA Model 2 


Intercept J3 Q (SE(J3 Q )) 


-7.85 (0.57)* 


-7.51 (0.51)* 


-2.53 (0.12)* 


-2.33 (0.16)* 


Number of classes taught by tenure- 
track faculty /?, (SE(/3 l )) 


0.57 (0.05)* 




0.15 (0.01)* 




Number of classes taught by 
contingent instructors fi 2 (SE(fi 2 )) 
Number of classes a student took 


0.58 (0.04)* 


0.58 (0.04)* 


0.15 (0.01)* 


0.14(0.01)* 


Female /3 4 (SE{P 2 )) 


-0.05 (0.11) 


-0.04 (0.11) 


0.17(0.03)* 


0.17(0.03)* 


Non-Caucasian /3 4 (SE(fi 4 )) 


-0.08 (0.16) 


-0.08 (0.16) 


-0.05 (0.04) 


-0.06 (0.04) 


State resident fi 5 (SE(fi 5 )) 


0.75 (0.11)* 


0.75 (0.11)* 


-0.00 (0.02) 


-0.00 (0.02) 


High School GPA J3 6 (SE(J3 6 )) 


0.76 (0.13)* 


0.67 (0.13)* 


0.69 (0.02)* 


0.67 (0.03)* 


ACT j3 7 (SE(j3 7 )) 


0.04 (0.02)* 


0.04 (0.01)* 


0.05 (0.00)* 


0.05 (0.00)* 


Member of a fraternity or sorority 

MSEW) 


0.97 (0.13)* 


0.97 (0.13)* 


0.08 (0.03)* 


0.08 (0.03)* 


Student-level (se(s* 0 )) 






0.42 (0.01)* 


0.42 (0.01)* 


Faculty-level Sp (se(s 2 m )) 






0.66 (0.09)* 


0.66 (0.09)* 


Chi-square (df=l), H () : /?, - P 2 = 0 


0.10 




0.14 




Deviance (MCMC) 


2541.81 


2540.70 


7704.66 


7703.61 


Deviance Information Criterion (DIC) 


2550.93 


2548.40 


7953.40 


7952.14 



*Significant at the 5% alpha level 
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FIG. 1 . Student-faculty level versus student level relations between instructor type and grade 
performance 
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FIG. 2. Grades nested within students and instructors: A cross-classified data 
structure. 
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FIG. 3. Classification diagrams for cross-classified and multiple membership 
models. 




Cross-classified model Multiple membership model 



46 



AIR 2010 Forum - Chicago, IL 













Running Head: CONTINGENT INSTRUCTORS AND STUDENT OUTCOMES 



FIG. 4. Expected probabilities of getting an “A”, a “B” or higher, and a “C” or higher for a 
typical student taught by a typical faculty with one standard deviation whiskers by faculty and by 
students. 
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FIG. 5. Residuals for 553 faculty members* 




*Based on equal weights unconditional model 
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